The methodological nettle: ICT and student achievement

A major challenge for researchers and educators has been to discern the effect of ICT use on student learning outcomes. This paper maps the achievements in Year 10 Science of two cohorts of students over two years where students in the first year studied in a traditional environment while students in the second took part in a blended or e-learning environment. Using both quantitative and qualitative methods, the authors have shown that ICT, through an e-learning intervention, did improve student performance in terms of test scores. They have also shown that this improvement was not global with the results for previously high-performing female students tending to fall while the results for lower-achieving boys rose. There was also a seeming mismatch between some students’ affective responses to the new environment and their test scores. This study shows the complexity of ICT-mediated environments through its identification and description of three core issues which beset the credibility of research in ICT in education. These are (1) ICT as an agent of learning, (b) site specificity, and (c) global improvement. Introduction Educators and researchers remain enthusiastic about the use of information and communication technology (ICT) in teaching and learning, even though the evidence supporting its benefits remains inconclusive (Liu, 2004; Reynolds, Treharne & Tripp, 2003; Underwood, 2004; Wellington, 2005). The most commonly found outcome is consistently one of ‘no significant difference’ (No Significant Difference Phenomenon, 2007), or an inability to isolate ICT as an independent variable. The ‘big’ question of research in ICT in teaching and learning, or e-learning, is what impact, if any, does it have on student outcomes. Reynolds et al (2003) asked ‘Where is the evidence that the ICT improved the pupils’ performance?—a methodological nettle that the IMPACT2 report research team noted as being outside their remit to grasp’ (p. 156). British Journal of Educational Technology (2008) doi:10.1111/j.1467-8535.2007.00790.x © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA. This paper will ‘grasp the nettle’ by considering, through both quantitative and qualitative means, student performance in a blended e-learning environment in junior secondary science classes, where web-based learning materials were seamlessly ‘blended’ with traditional classroom activities (see Chandra, 2004). It will also attempt, through this case, to identify the fundamental problems—here called the ‘nettles’—that hamper continuing research in ICT in education in its goal to answer broad questions and the establish universal tenets. Background The study reviewed in this paper was conducted at a co-educational state secondary school in Queensland, Australia (see Chandra, 2004). It considered successive cohorts of Year 10 science students (15–16 years of age) over 2 years. Both Cohort 1 and Cohort 2 undertook the same semester program in terms of subject content, but one, Cohort 2, studied in a blended e-learning environment. This program is summarised in Table 1. Cohort 1 served as the control in the study and is referred to as the Traditional group (N = 210), while Cohort 2, the focus of this paper, will be referred to as the Blended group (N = 232). Both cohorts were of comparable size, gender balance, and evidenced a similar mix of ethnic backgrounds. The following discussion will, in turn, analyse the performance and learning experiences of these students using both quantitative and qualitative methods. Quantitative analysis In Term 1, both cohorts studied a chemistry unit that focused on the topics of electrochemistry and consumer science.They were taught using traditional pedagogy and were assessed in similar ways. In Term 2, students, as is common in science subjects, undertook a unit in physics that covered road science and space. The expectation, based on prior experience, was that students would generally achieve lower test scores in this unit as it was perceived to be more demanding. An exception to this was the lower-achieving boys who had, in the past, tended to ‘do better’ in this unit compared to the rest of their cohort. One of the teachers commented that ‘perhaps the [Physics] unit involved concepts that boys could relate to—such as speed, acceleration, tyres, [and] braking’. All students sat, at the end of each term, for tests developed within the school. Performance data was therefore available for each cohort at the end of each term/unit and made possible comparisons between the cohorts and the pedagogical approaches. The Table 1: Semester course of study Unit Cohort 1 (N = 210) Cohort 2 (N = 232) Term 1 Chemistry Traditional Traditional Term 2 Physics Traditional Blended 2 British Journal of Educational Technology © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. means from the tests were compared using a paired sample t-test, and further analysis was done by rank ordering the boys’ and girls’ results from the Term 1 unit and then dividing these into quartiles. The results from students in each quartile were then compared with the results from the Term 2 unit using a paired sample t-test. These results are presented in Tables 2–5. Table 2 presents the overall test results for boys from both the Traditional (Cohort 1) and Blended (Cohort 2) learning environments. The boys in the Traditional group (Cohort 1) obtained a lower mean in the test after the physics unit (Pre-M > Post-M). This result was reversed for the boys (Pre-M < Post-M) in the Blended group (Cohort 2), and the difference between the means for the Blended group was statistically significant (p < 0.01). The SD for the Blended group was also lower (Pre-SD = 21.3, Post-SD = 17.5). These results suggest that the ICT-based interTable 2: Boys’ test results Group N Pre-M Post-M M-Diff Pre-SD Post-SD SD-Diff df Traditional 110 66.4 64.1 -2.3 23.1 21.8 -0.7 109 Blended 132 70.1 75.2 5.1** 21.3 17.5 -3.8 131 N = sample size; Pre-M = mean in the test before the physics unit; Post-M = mean in the test after the physics unit; M-Diff = Post M – Pre-M; Pre-SD = SD in the test before the physics unit; PostSD = SD in the test after the physics unit; SD-Diff = Post-SD Pre-SD. **p < 0.01. Diff, difference; df, degrees of freedom. Table 3: Boys’ test results based on quartiles Quartile N Pre-M Post-M M-Diff Pre-SD Post-SD SD-Diff df Traditional Group (Cohort 1) 1 28 92.1 85.5 -6.6** 4.1 9.3 5.2 27 2 28 78.5 72.6 -5.9* 5.2 12.5 7.3 27 3 27 59.4 54.4 -5.0* 5.5 14.5 9.4 26 4 27 34.4 42.6 8.2* 13.5 21.1 7.6 26 Blended Group (Cohort 2) 1 33 91.2 90.1 1.1 3.9 6.4 3.1 32 2 33 81.6 83.7 2.1 3.3 7.6 4.3 32 3 33 70.8 68.5 2.3 5.1 13.9 8.8 32 4 33 39.0 56.4 17.4** 14.1 16.6 2.5 32 N = sample size; Pre-M = mean in the test before the physics unit; Post-M = mean in the test after the physics unit; M-Diff = Post-M Pre-M; Pre-SD = SD in the test before the physics unit; PostSD = SD in the test after the physics unit; SD-Diff = Post-SD Pre-SD. *p < 0.05; **p < 0.01. Diff, difference; df, degrees of freedom. ICT and student achievement 3 © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. vention impacted on the results and the smaller SD suggests that the spread was narrower. Table 3 shows the performance of boys across quartiles. Table 3 shows that the differences in the means for boys in the Traditional group across the quartiles were negative except for Quartile 4, where it was positive. All differences were statistically significant (p < 0.05). These results suggest that, with the exception of the ‘at risk’ boys (a mark less than 50%) in Quartile 4, students found the physics unit more difficult than the chemistry unit taught in the previous term. The means obtained by students in the Blended group (Cohort 2) were comparatively different. While the differences in all means (M-Diff) across all quartiles were positive, the difference was statistically significant only for students in Quartile 4 (p < 0.01). The difference in the means for these students was almost double (M-Diff = 17.4) than that of the students in Quartile 4 of the Traditional group (M-Diff = 8.2). The differences in the means suggest that, in comparison to the Traditional group, boys in the Blended Table 4: Girls’ test results Group N Pre-M Post-M M-Diff Pre-SD Post-SD SD-Diff df Traditional 100 77.4 68.0 -9.4** 16.4 20.1 3.5 99 Blended 100 73.5 71.6 -1.6 18.7 18.0 -0.7 99 N = sample size; Pre-M = mean in the test before the physics unit; Post-M = mean in the test after the physics unit; M-Diff = Post M Pre M; Pre-SD = SD in the test before the physics unit; PostSD = SD in the test after the physics unit; SD-Diff = Post-SD Pre-SD. **p < 0.01. Diff, difference; df, degrees of freedom. Table 5: Girls’ test results based on quartiles Quartile N Pre-M Post-M M-Diff Pre-SD Post-SD SD-Diff df Traditional group 1 25 93.5 91.8 1.7 4.6 4.8 0.2 24 2 25 86.0 77.5 -8.5** 8.2 5.3 -2.9 24 3 25 69.0 62.0 -7.0** 8.5 4.5 -4.0 24 4 25 61.4 41.5 -19.9** 15.5 10.5 -5.0 24 Blended group 1 25 93.5 85.8 -7.7** 3.9 9.6 5.7 24 2 25 83.4 77.5 -5.9* 3.0 13.1 10.1 24 3 25 69.9 67.4 2.5 4.5 13.4 8.9 24 4 25 47.3 55.8 8.5* 12.4 18.5 6.1 24 N = sample size; Pre-M = mean in the test before the physics unit; Post-M = mean in the test after the physics unit; M-Diff = Post M Pre M; Pre-SD = SD in the test before the physics unit; PostSD = SD in the test after the physics unit; SD-Diff = Post-SD Pre-SD. *p < 0.05; **p < 0.01. Diff, difference; df, degrees of freedom. 4 British Journal of Educational Technology © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. group generally performed better across all four quartiles. The boys in Quartile 4 seemed to have benefited the most from the intervention. The statistical significance in the difference of the means also changed from p < 0.05 to p < 0.01. The performance of the girls in the Traditional group (Cohort 1) was similarly compared with the results obtained by the girls in the Blended group (Cohort 2). The overall findings are reported in Table 4. For the girls in the Traditional group, M-Diff was negative because the mean obtained after the completion of the physics unit (Post-M) was lower than the mean from the test done before this unit (Table 3). This difference was also statistically significant (p < 0.01). The difference in the mean for the Blended group was negative, smaller and not statistically significant. As with the boys (Table 2), the girls in the Blended group also performed better in the test on the physics unit. The performance of the girls in each quartile is presented in Table 5. Table 5 shows that the differences in the means for the girls in the Traditional group across Quartiles 2, 3 and 4 were negative and statistically significant (p < 0.01). The positive difference in the mean for girls in Quartile 1 was due to their higher test scores on the physics unit.The girls in the Blended group in the first and second quartiles had mean differences of -7.7 and-5.9 respectively. Both differences were statistically significant at p < 0.01 (Quartile 1) and p < 0.05 (Quartile 2). When compared to the Traditional group, the level of significance changed for students in Quartile 2—from 0.01 to 0.05— which suggested that the Blended group achieved a comparatively higher mean in the physics test. Similarly, the students in Quartiles 3 and 4 also achieved higher means that produced positive results for M-Diff—both these results paralleled the results obtained by boys in these quartiles (see Table 2). The difference in the mean for Quartile 4 was statistically significant (p < 0.01). The difference in the means of test scores for the girls in Quartile 1 were negative and statistically significant (p < 0.05). On the basis of these results, it would appear that the girls in the first quartile benefited the least by this initiative. On the other hand, the girls in Quartiles 3 and 4 (equating to 50% of all females) appeared to have performed better as a result of the ICT intervention. A consideration of all student results (Tables 2–5) reveals that the test scores for the majority of students in the Blended group improved for the physics unit, but did they not improve at the same rate. The expected fall (negative change) did not occur in the overall results for the boys (M-Diff = 5.1, Table 2), and was significantly reduced for the girls (M-Diff = -1.6 cf. M-Diff = -9.4, Table 4). The simple conclusion to be drawn from this is that the use of ICT does improve student performance. The more complex conclusion is one that adds the caveat that this improvement is not equal, and that the impact of ICT is evidenced differently for different groups. Qualitative analysis To identify more qualitative impacts of the ICT intervention described in this paper, comments from selected students (N = 14) from the Blended group will be analysed. This ICT and student achievement 5 © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. comprises of nine instances where student test scores improved (m = 6, f = 3), four where test scores fell (m = 1, f = 3), and, finally, one female whose score remained static (98%). Details of the selected instances where test scores improved are presented as Table 6. The first four students cited, namely Gary, Murray, Mike and Donald, belonged to the fourth (lowest) quartile (Table 3) who had, over time, shown improved test scores in this unit. This group had shown heightened improvement through blended learning, (M-Diff = 8.2 cf. M-Diff = 17.4). These boys’ comments indicate parallel heightened levels of engagement and enjoyment. The greatest improvement was by Gary (+55%), a migrant, for whom English was a second language. Similar improvement, but to a lesser degree (22%), was seen in Mike’s results, another English as Second Language student. This suggests that the additional e-learning scaffolds supported students for whom standard classrooms were confusing and conceptual understandings were impeded by language and cultural issues. The majority of students presented in Table 6 have commendably moved, based on test scores, from a failing ( 49%) to a passing grade ( 50%). The fifth and sixth cases, that is, Hannah and Sally, showed improvements of 20% (from 32 to 52%) and 14% (from 36 to 50%) respectively. This is contrary to the pattern of results for girls in both the Traditional and Blended groups. Hannah credited this to her taking control of her learning, and by denigrating the traditional science classroom as being ‘not interesting’ and where teachers ‘mumble’ and students ‘fall asleep’. Sally commented on ‘being bored’ by classroom ‘busy work’ of writing and listening. Both found the blended environment more interesting and easier to understand and indirectly made reference to independence and student-centred learning. The improvement for high achievers, namely, Dolly, Harold and Peter, is intriguing, and the students’ own arguments highlight the affordances of e-learning, that is, differing representations of information, animations and interactivity, and the possibility for extension and capacity for customisation. These students valued the ability to revisit materials as needed. It is of interest to compare the outcomes for Dolly and Sally whose results had improved by 14%. The salient difference is that Dolly’s results moved from 86 to 100% while Sally’s, more modestly, moved from 36 to 50%. What the girls shared, despite the contrast in achievement, was heightened enjoyment from the learning environment and the independence they were afforded. Both used the term ‘interesting’ and referred to the extra resources, namely diagrams and examples, they used to extend their own learning. Despite the ‘distance travelled’ in terms of test results, these students acknowledged the notion of working independently and at their ‘own pace’. The e-learning experience seemed positive for all above and beyond numerical test scores, and even those who could be labelled as low-achieving were metacognitively aware of their own learning and motivation to learn. 6 British Journal of Educational Technology © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. Table 6: Improved test scores (sorted by difference) Name Gender Result 1 (%) Result 2 (%) Difference (%) Comment Gary M 30 88 +55 I can learn in my own house and at any time I want ... I can learn at my own rhitim [rhythm] Murray M 40 76 +36 It was funnier [more fun] than sitting in class and listening ... I enjoy the lessons because it helps me learn more. Mike M 32 54 +22 You can work at your own pace and the work is set up in a clear way you can understand ... web lessons also included a test at the end to see how you were going. Donald M 26 48 +22 I am not easily distracted on the Internet lessons. In class I am distracted easier. Instead of listening to the teacher and nearly falling asleep, you can just read off the computer ... I think I have learnt better off the Internet. Hannah F 32 52 +20 It is easier to understand and comprehend because you can read it at your own pace and you don’t have to listen to a teacher mumble on ... normal classes are not interesting when teachers are talking because some students fall asleep. Sally F 36 50 +14 We do not have to do as much writing. We do not have to listen to the teacher as much and get bored by it. And the website is easy to use and I learn much more because I find it a lot more interesting ... We can do the work ourselves and website has examples we can use. I think it is great. Dolly F 86 100 +14 You can go back again and again until you understand it. It is all at your own pace and more interesting than normal work ... there are diagrams and extra information ... it was a good change from the classroom and you get one on one experience. Harold M 86 98 +12 It is more interesting than learning out of a textbook as the web page is tailored to the work we are doing in class ... it helps me learn more by the diagrams. Peter M 74 80 +06 Animations make understanding of the work easier ... There is also some stuff on the Internet which is not in class. It is also more interactive and has good content. M, male; F, female. ICT and student achievement 7 © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. Details of instances where students’ results fell are presented as Table 7. It is important to recall that the girls in the first quartile benefited the least by the ICT intervention (Table 4). This is exemplified by Betty, Cindy and Nelly, whose comments, despite their reduced numerical result, are positive in tone. They enjoyed the change in pedagogy and seemed either unaware or unconcerned about their falling grades. Their test scores seemed to be disconnected from their learning, as noted by Nelly, who despite a fall of 18%, offered that ‘you become open-minded and look into new opportunities of learning’. Tom (fall of 10%) voiced a rarely noted criticism of e-learning by offering (elsewhere in the survey form) that ‘you can learn without understanding the work ... in parrot fashion’. He, in diametric contrast to many of his peers, saw the teacher and traditional lesson as being the core to teaching and learning in science. He described e-learning as Table 7: Diminished test scores Name Gender Result 1 Result 2 Difference Comment Betty F 90 82 -8 You can read through the information and find out what you do not understand ... and we also got out of the room and did not have to watch the teacher. Tom M 98 88 -10 Web pages are presented in a manner that is easy to follow ie, you can re-read what you do not understand. It is put into a way where the content is arranged in appropriate categories. It also helps you find weaknesses through online tests. I am sick of a lecture in class, they are important but computers should be used to add variety to the subject. They are a good way to revise and learn but should not take the importance of a class lesson. Cindy F 88 72 -16 The approach is far more interesting than learn[ing] out of a textbook and most of the time, it’s much easier to comprehend. If you do not understand something, you can send your teacher an email. Nelly F 92 74 -18 You become open-minded and look into new opportunities of learning. I also like learning off the Internet and taking sheets back home to learn off, however, sometimes you are not focused on the task. M, male; F, female. 8 British Journal of Educational Technology © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. being complementary, that is, ‘a good way to revise and learn but should not take the importance of a class lesson’. Criticism can also be noted in Nelly’s caveat that ‘sometimes you are not focused on the task’, perhaps acknowledging the inherent difficulties with the convergent rather than divergent nature of e-learning. Details of the student whose results remained static are presented as Table 8. Alison was high-achieving, and e-learning did not appear, based on test scores, to affect her learning outcomes. However, through her comments, it becomes apparent that she believed she learned more and could identify particular scaffolds, namely tests and worksheets, that contributed to her understanding. The results of other girls in Quartile 1 were diminished, and while it could be contended that Alison adapted more quickly to the new environment, it could also be more contentiously suggested that she imposed the habitus of a science textbook and low-level memorisation on to the new environment. This can be drawn, tenuously, from her reference to ‘worksheets’ and processes such as ‘revise and study’. Nelly (see Table 7) had suggested a similar technique, that is, ‘taking sheets back home to learn off’, but had less success than Alison. These techniques can be contrasted to previously cited reference to independence and investigation, divergence over convergence, and of extending understanding through media such as animations and simulations. Following the quantitative analysis, the simple conclusion was that e-learning does improve student performance. The qualitative analysis allows the extension of this conclusion to include enhanced metacognition, student engagement and descriptors of learning, which resonate with notions of cognitive apprenticeship (Collins, Brown & Newman, 1989). The more complex conclusion from the quantitative analysis added the caveat that this improvement was not equal, and that the impact was evidenced differently in different groups. The qualitative analysis suggests a similar but not parallel experience. Students’ adoption and comfort with e-learning differed, and this, intriguingly, seemed to be irrespective of numerical scores. Table 8: Static test results Name Gender Result 1 Result 2 Difference Comment Alison F 98 98 0 You can go back over the work again as many times as you like. Having the Internet worksheets from class lessons and the multiple choice tests at the end of each lesson help you revise and study ... and learn the work more. M, male; F, female. ICT and student achievement 9 © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. Nettles—where are thy stings? The seeming contradictions emerging through the quantitative and qualitative analysis here serve to reveal the core issues, the stings of the methodological nettles, that beset the credibility of research in ICT education and stand to limit its achievement of broadly generalisable tenets. These are identified here as (1) ICT as an agent of learning, (2) site specificity, and (3) global improvement. ICT as an agent of learning Isolating the e-learning environment from other potential variables is difficult, and we should heed the cautionary warning in ‘ascribing to ... technology powers it does not possess’ (Snyder, 2000, p. 111). Similarly, if we accept that it is not technology, but its facilitation of altered pedagogy (after Gunawardena & McIsaac, 2004) that has wrought change, then we have a smaller less tangible target to which to attribute causality. There is a complexity in e-learning (Liu, 2004) that can be masked by the drawing of simplistic conclusions. The positive impact on student learning reported in this paper might, for instance, arise from the renewed enthusiasm of staff and students or the ‘halo’ effect of being part of a research study. Students were reportedly more engaged and subsequently more motivated to learn. One teacher, following the ICT intervention, commented that: ... students were very different in the computer laboratories. They were all actively engaged and this applied to even those who had a history of not doing so in normal classrooms. One of the key aspects was their engagement was that the task was targeted at their level and consequently was doable and they were aware of what was expected. Aimless searches on the web did not occur. Interestingly, not all students accepted e-learning unreservedly (see Table 7, Tom and Nelly). Some students, like some teachers, are technophobic and hold mental models of schooling as teacher-centred and instructivist. While references to ‘sleeping in class’ and the characterisation of teachers as tedious pedagogues seem stereotypical of adolescence and represent expected rather than heartfelt responses, the motif of enjoying novelty and change of routine should not be dismissed. Dolly (who scored 100%) offered ‘it was a good change from the classroom’ (Table 6), while Betty (Table 7), with the excitement of a released prisoner, commented that they ‘got out of the room’. The most critical finding is that students appeared to be aware of the differences between traditional and e-learning environments, and seized the opportunity to work and learn in a different space. The cited teacher’s observation of student engagement reinforces the reported experience and also describes a story of altered classroom dynamics where the teacher focuses on the science and on mentoring rather than instructing. It may be that the change in pedagogy rather than the technology itself that has influenced student outcomes. The indicators reported in this paper, that is, improved test scores and heightened student engagement, support the contention that e-learning can be beneficial to student learning. The critical and unanswered question is what particular characteris10 British Journal of Educational Technology © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. tics of the environment were the catalysts for change. The students’ comments indicates that what they most enjoyed was the control and self-regulation they were allowed. The commonality of student comments (Tables 6–8) may reveal the characteristics that have wrought improvements rather than the agency of the technology itself.